supervised_learning/Random Forests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification trees\n",
    "_Learn by splitting the feature space_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Decision boundaries\n",
    "![Decision Tree](images/dt.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Feature Splits\n",
    "![Decision Tree](images/dt-boundaries.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Tree\n",
    "![Decision Tree](images/dt-tree.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hunt's Algorithm\n",
    "\n",
    "* Take one feature at a time\n",
    "    * If a node is pure\\*, it is a leaf node\n",
    "    * If a node is impure, create new internal node\n",
    "        * Consider all binary splits of given feature space\n",
    "            * For each split, measure impurity\n",
    "            * Take the split with smallest impurity as node value\n",
    "            \n",
    "Iterate until convergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Stopping conditions\n",
    "\n",
    "* Node is pure\n",
    "* Node has small impurity\n",
    "* Node has small number of datapoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![gini](images/gini.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Impurity measures\n",
    "\n",
    "All are measures of misclassification\n",
    "\n",
    "* $Entropy(t) = -\\sum_{i=1}^{C}p(i)log p(i)$\n",
    "\n",
    "* $Gini(t) = 1 - \\sum_{i=1}^{C}p(i)^2$\n",
    "\n",
    "* $Error(t) = 1-{max}_{i \\in C}[p(i)]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from tree_util import draw_tree_to_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Example - Iris\n",
    "![Iris](images/iris.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "iris = load_iris()\n",
    "clf = DecisionTreeClassifier().fit(iris.data, iris.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "draw_tree_to_file(clf, iris['feature_names'], \"images/dt-iris.dot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!dot -Tpng images/dt-iris.dot -o images/dt-iris.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![decision-tree](images/dt-iris.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Random Forests\n",
    "_Bootstrap with many decision trees_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# One tree is not enough\n",
    "\n",
    "Running decision tree on the _same_ dataset will give the _same_ result.\n",
    "\n",
    "This will miss less informative features.\n",
    "\n",
    "![rf](images/random-forest.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Algorithm\n",
    "\n",
    "* Select a subset of features\n",
    "* Run decision tree algorithm\n",
    "\n",
    "# Prediction - bagging\n",
    "\n",
    "* Take majority vote"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Summary\n",
    "\n",
    "## Pros\n",
    "\n",
    "* Highly non-linear\n",
    "* Interpretable\n",
    "\n",
    "## Cons\n",
    "\n",
    "* Easy to overfit\n",
    "* Expensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Further reading\n",
    "\n",
    "* Introduction to data mining by Tan, Steinbach, Kumar\n",
    "    * [Chapter 4](https://www-users.cs.umn.edu/~kumar/dmbook/ch4.pdf)\n",
    "* Random Forests, Leo Breiman. 2001\n",
    "    * [Machine Learning, Springer](http://link.springer.com/article/10.1023%2FA%3A1010933404324)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#import the classifiers\n",
    "from sklearn.metrics import roc_curve, roc_auc_score\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "#import the functions to use the dataset\n",
    "from pathogenicity_predictor import prepare_variants, concat_training_data, partition_into_training_and_test, plot_line_graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data, feature_names = prepare_variants('../data/variants.json.gz')\n",
    "variants, labels = concat_training_data(data)\n",
    "training_vars, test_vars, training_labels, test_labels = partition_into_training_and_test(variants, labels, 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a decision tree!\n",
    "tree_classifier = DecisionTreeClassifier().fit(training_vars, training_labels)\n",
    "\n",
    "# Run random forest classification!\n",
    "rf_classifier = RandomForestClassifier().fit(training_vars, training_labels)\n",
    "\n",
    "# Get probabilities!\n",
    "pathogenicity_probs = rf_classifier.predict_proba(training_vars)[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature importances\n",
    "\n",
    "Decision trees and random forests can tell you how many times a feature is used in the classifier and how much information is gained by using the splits for this feature. This amalgamates into feature importances vector that you can see with `classifier.feature_importances_`.\n",
    "\n",
    "# Question 1\n",
    "Do the results make sense?\n",
    "\n",
    "# Question 2\n",
    "How does this compare between between a random forest and one decision tree?\n",
    "\n",
    "# Question 3\n",
    "Build a $ROC$ curve for the random forest classifier. Is it a good classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  },
  "toc": {
   "toc_cell": false,
   "toc_number_sections": false,
   "toc_threshold": 6,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}